feat(low-code): support request window splitting on partitioned streams

[Kaizer (kaizerch)]

Overview

👉 TL;DR: Streams that sync each account separately, like Google Ads customers or TikTok advertisers, can now automatically split a date window the API rejects instead of failing the sync. Only the account whose request was rejected gets split; other accounts, and the progress saved for them, are untouched. Specifically, ConcurrentPerPartitionCursor gains split_request_window, so the factory no longer rejects request_window_splitting on a stream with a partition router.

Builds on #1171 (released in 7.32.0), which added SPLIT_REQUEST_WINDOW for unpartitioned streams only:

• feat: add SPLIT_REQUEST_WINDOW action for adaptive request-window splitting #1171

Unblocks https://github.com/airbytehq/airbyte-internal-issues/issues/17176 (TikTok Marketing) and https://github.com/airbytehq/airbyte-internal-issues/issues/17179 (Google Ads), part of epic https://github.com/airbytehq/airbyte-internal-issues/issues/17173.

Changes

• ConcurrentPerPartitionCursor.split_request_window(stream_slice, min_split_window): delegates to one state-free ConcurrentCursor, built from the stream's own _cursor_factory on first use and cached under the existing _lock. The children keep the slice's partition and extra_fields.
• The factory's hasattr(cursor, "split_request_window") check now passes for partitioned streams, so no factory wiring changes. The "unsupported cursor" error no longer says partition routers are unsupported. Every other load-time check applies to partitioned streams unchanged.
• Tests:
  • ConcurrentPerPartitionCursor: halves keep partition and extra_fields, a 1-day window returns None, min_split_window is honored, and the splitting cursor is built once and only on first use.
  • Factory: the "partition router is rejected" test becomes "partitioned stream builds and binds the per-partition cursor's splitter", and the reworded error is asserted.
  • End to end, two accounts with only one account's window rejected through a real http_codes: [400] filter. That account is read as two halves, the other once, and each keeps its own cursor in the final per-partition state. It runs with start_time_option/end_time_option, and with a list-form partition_router plus stream_interval interpolation.
  • The same successful read past SWITCH_TO_GLOBAL_LIMIT: it splits and checkpoints one global cursor. Only this success case runs in global-cursor mode; the still-rejected case below runs per partition.
  • A partly read account: its first half is read, its second half is rejected again at min_split_window. The stream fails with transient_error, that account's cursor stays at its start instead of moving to the record already read, and the other account is still checkpointed.

Review Spotlight

Reviewers with limited time, please review first:

• concurrent_partition_cursor.py#L682-L708: the new split_request_window and why one state-free cursor is enough
• model_to_component_factory.py#L4368-L4372: the reworded error, the only factory change
• test_concurrent_declarative_source.py#L5396-L5519: the end-to-end per-partition, global-cursor and still-rejected tests

Not in scope: custom retrievers

This PR doesn't change custom retrievers. A CustomRetriever that declares request_window_splitting in YAML still fails at load ("Subcomponent creation has not been implemented for 'RequestWindowSplitting'"), and the factory doesn't hand it a splitter. Google Ads, the only custom-retriever consumer, does the wiring in Python, not YAML: GoogleAdsRetriever gets a cursor field (the factory already passes cursor= to every retriever) and sets request_window_splitter and request_window_splitting in __post_init__. I checked that pattern against this branch with a scratch custom retriever on a partitioned read: only the rejected account split, and both accounts' cursors were correct.

Risks

• No existing connector changes behavior. Before this PR, declaring the block on a partitioned stream failed at load time, so only a connector that opts in after this release is affected. The only connector using the block today, PayPal Transaction, is unpartitioned and takes the unchanged ConcurrentCursor path.
• Unchanged from 7.32.0: split children aren't clamped again to the cursor's start or end, and records already emitted before a rejection are re-emitted when the window is split and re-read (logged as a warning). Both behave exactly as they do for unpartitioned streams.

Post-merge actions

• Release as 7.33.0 (minor: new capability, no breaking change). Publishing the tag also publishes the source-declarative-manifest image; confirm airbyte/source-declarative-manifest:7.33.0 appears on DockerHub.
• Open the TikTok Marketing (manifest only) and Google Ads (wires the splitter in its custom retriever) adoption PRs against the release.

Test plan

• All 10 new or changed test cases pass at e0371aac and fail on main. With only the concurrent_partition_cursor.py change reverted, 9 of them fail. The tenth, test_given_no_incremental_sync_and_request_window_splitting_then_raise, asserts the reworded error message, so it fails only when the message is reverted too.
• poetry run pytest unit_tests -m "not slow and not requires_creds": 4,917 passed, 2 failed. Both failures are Docker image-build tests (test_docker_image_build_and_spec, test_docker_image_build_and_check) that failed locally because Docker wasn't running; unrelated to this change.
• ruff check, ruff format --check, and mypy on the two changed source files pass.

Design notes: why one state-free cursor instead of each partition's own

Show/Hide Content

• ConcurrentCursor.split_request_window is a pure function of the slice's boundaries and the cursor's granularity and date format. It never reads cursor state. Every per-partition cursor comes from the same _cursor_factory, so they would all return the same split.
• Looking up the partition's own cursor would break after the switch to a global cursor (SWITCH_TO_GLOBAL_LIMIT), when _ensure_partition_limit may already have evicted it.
• The cursor is built on first use, so streams that never split pay nothing, and tests that hand _cursor_factory.create a fixed side_effect list keep working. This uses the same create(stream_state={}, runtime_lookback_window=None) call as get_cursor_datetime_from_state.
• observe and close_partition need no change: SimpleRetriever already re-associates records read from split children with the original slice, and a partition only closes after all of its children succeed.
• split_request_window is not added to the Cursor ABC. A default there would let FinalStateCursor pass the factory's hasattr check.

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@kaizerch/feat/17179-17176-cdk-request-window-splitting-support-google-ads-tiktok#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch kaizerch/feat/17179-17176-cdk-request-window-splitting-support-google-ads-tiktok

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[Kaizer (kaizerch)]

/prerelease

[github-actions]

PyTest Results (Fast)

4 930 tests  +8   4 918 ✅ +8   10m 19s ⏱️ -32s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 1d62b0f. ± Comparison against base commit d5536bc.

This pull request removes 1 and adds 9 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_given_partition_router_and_request_window_splitting_then_raise

unit_tests.sources.declarative.incremental.test_concurrent_perpartitioncursor ‑ test_given_min_split_window_when_split_request_window_then_stop_at_min_split_window
unit_tests.sources.declarative.incremental.test_concurrent_perpartitioncursor ‑ test_given_one_day_partition_window_when_split_request_window_then_none
unit_tests.sources.declarative.incremental.test_concurrent_perpartitioncursor ‑ test_given_partition_window_when_split_request_window_then_halves_keep_partition_and_extra_fields
unit_tests.sources.declarative.incremental.test_concurrent_perpartitioncursor ‑ test_split_request_window_builds_one_state_free_cursor_on_first_use
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_given_partition_router_and_request_window_splitting_then_split_with_per_partition_cursor
unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_given_partitioned_stream_over_switch_to_global_limit_when_window_is_rejected_then_split
unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_given_partitioned_stream_when_one_partition_window_is_rejected_then_split_only_that_partition[interpolated_window]
unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_given_partitioned_stream_when_one_partition_window_is_rejected_then_split_only_that_partition[request_options]
unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_given_split_window_still_rejected_when_read_then_partition_is_not_checkpointed

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 933 tests   4 921 ✅  15m 47s ⏱️
    1 suites     12 💤
    1 files        0 ❌

Results for commit 1d62b0f.

♻️ This comment has been updated with latest results.

[Anatolii Yatsuk (tolik0)]

The description still covers the first commit, and it becomes the release notes. Could you update it?

• Drop the factory registration, the create_default_stream pass-through and the CustomRetriever tests. They aren't in the diff, and a CustomRetriever declaring the block still fails at load.
• Drop the "Known exception" section and the Risks bullet about SimpleRetriever subclasses.
• Test plan: 9 of the 10 tests fail without the change, not all 10. The failure case also isn't run in global-cursor mode.
• For Google Ads, say the wiring is in Python (a cursor field, plus the splitter set in __post_init__), not in YAML.
• Pin the Spotlight links to e0371aac. The factory ones now point at unrelated code.

[Kaizer (kaizerch)]

The description still covers the first commit, and it becomes the release notes. Could you update it?

• Drop the factory registration, the create_default_stream pass-through and the CustomRetriever tests. They aren't in the diff, and a CustomRetriever declaring the block still fails at load.
• Drop the "Known exception" section and the Risks bullet about SimpleRetriever subclasses.
• Test plan: 9 of the 10 tests fail without the change, not all 10. The failure case also isn't run in global-cursor mode.
• For Google Ads, say the wiring is in Python (a cursor field, plus the splitter set in __post_init__), not in YAML.
• Pin the Spotlight links to e0371aac. The factory ones now point at unrelated code.

Thanks, updated. The description now matches e0371aa only: the custom-retriever wiring and tests are gone, and it says a CustomRetriever declaring the block still fails at load, with Google Ads wiring the splitter in Python. Test plan is corrected: all 10 cases fail on main; with only the cursor change reverted, 9 fail, and the tenth asserts the reworded error message. Only the success case runs in global-cursor mode. Spotlight links are pinned to e0371aa.

[Kaizer (kaizerch)]

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/36925038601