fix: prevent record filters from prematurely stopping PageIncrement pagination

[Daryna Ishchenko (darynaishchenko)]

Summary

PageIncrement stops paginating when last_page_size < page_size, but SimpleRetriever._read_pages computes last_page_size from records after the RecordSelector runs (post record-filter). So a record filter (e.g. client-side incremental filtering) that drops records from a full API page causes pagination to stop early — and if an entire page is filtered out, the sync stops entirely even though more pages exist.

OffsetIncrement was already fixed for this exact issue by accepting an optional extractor that re-counts the raw (pre-filter) records from the response. This PR mirrors that approach for PageIncrement:

• PageIncrement gains an optional extractor: Optional[RecordExtractor] field; when set, next_page_token recomputes last_page_size = len(extractor.extract_records(response)) before applying the stop condition.
• ModelToComponentFactory.create_page_increment now builds that extractor from the stream's record_selector.extractor model (same pattern as create_offset_increment, using the extractor_model already threaded through create_default_paginator).

Behavior is unchanged for direct PageIncrement instantiations without an extractor.

Requested by Daryna Ishchenko (Daryna Ishchenko (@darynaishchenko)).

Link to Devin session: https://app.devin.ai/sessions/ef2cfea26843460dafd703d1d4adaf29

Summary by CodeRabbit

• Bug Fixes

  • Improved page-based pagination to account for the number of records actually extracted from each response.
  • Pagination now correctly advances after full pages and stops after partial or empty pages.

• Tests

  • Added coverage for extracted-record pagination across full, partial, and empty response scenarios.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1784038497-page-increment-record-filter#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1784038497-page-increment-record-filter

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 131 tests  +4   4 119 ✅ +4   7m 42s ⏱️ -1s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit d3e2e20. ± Comparison against base commit dd9558c.

[github-actions]

PyTest Results (Full)

4 134 tests  +4   4 122 ✅ +4   12m 26s ⏱️ + 5m 11s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit d3e2e20. ± Comparison against base commit dd9558c.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a8c6b38b-c14a-43c9-a7ed-f536bc46066b

📥 Commits

Reviewing files that changed from the base of the PR and between dd9558c and d3e2e20.

📒 Files selected for processing (3)

• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/requesters/paginators/strategies/page_increment.py
• unit_tests/sources/declarative/requesters/paginators/test_page_increment.py

---

📝 Walkthrough

Walkthrough

PageIncrement now optionally uses an instantiated record extractor to calculate effective response page size. The component factory accepts extractor configuration, and tests cover continuation and stopping for full, partial, and empty extracted pages.

Changes

Page pagination

Layer / File(s)	Summary
Factory extractor wiring airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py	create_page_increment now accepts extractor configuration, creates an optional extractor component, and passes it to PageIncrement.
Extracted page-size pagination airbyte_cdk/sources/declarative/requesters/paginators/strategies/page_increment.py, unit_tests/sources/declarative/requesters/paginators/test_page_increment.py	PageIncrement derives page size from extracted records when configured, with tests for full, partial, and empty pages.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Requester
  participant PageIncrement
  participant RecordExtractor
  Requester->>PageIncrement: pass HTTP response
  PageIncrement->>RecordExtractor: extract records when configured
  RecordExtractor-->>PageIncrement: return extracted records
  PageIncrement-->>Requester: return next page token or None

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main fix: PageIncrement now uses raw record counts so record filters don't stop pagination early.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1784038497-page-increment-record-filter

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}