feat(low-code): add RateLimitedMultipleTokenAuthenticator component

[Daryna Ishchenko (darynaishchenko)]

Summary

Adds a new declarative authenticator, RateLimitedMultipleTokenAuthenticator, that generalizes source-github's MultipleTokenAuthenticatorWithRateLimiter so it can be reused by any connector during declarative migrations (context: https://github.com/airbytehq/airbyte-internal-issues/issues/16512).

Behavior (per outgoing request, via AuthBase.__call__):

• Requests are classified into named quota pools (quotas) via HttpRequestRegexMatchers; a pool with no matchers is the default (generalizes GitHub's hardcoded "graphql" in path REST-vs-GraphQL split).
• Each token tracks a local per-pool counter, seeded once per token from quota_status_source (e.g. GitHub GET /rate_limit) using remaining_path/reset_path/limit_path dpath extraction. Seeding is lazy (first request) and repeated after an exhaustion wait.
• Active token's counter is decremented; on exhaustion the authenticator rotates through tokens; when all are drained it sleeps until the earliest reset (heartbeat logs, floored at MIN_EXHAUSTION_WAIT = 5s so stale reset timestamps can't busy-loop the quota endpoint, bounded by max_wait_time, else raises transient_error).
• Proactive throttling: once every token drops below max(budget_min_reserve, budget_reserve_fraction * limit), delays of seconds_to_reset / total_remaining (capped at 10s) are injected for graceful degradation instead of a hard wall.

Thread safety (new vs. the source-github implementation, required for the concurrent CDK):

• Token selection + decrement happen atomically under an RLock; _acquire_call returns the token chosen under the lock and the request header is built from that value, so a request is always signed with the token whose counter was decremented (no TOCTOU with concurrent rotation). All sleeps occur outside the lock.
• Double-checked re-seed via a separate _refresh_lock so only one thread refreshes counters after an exhaustion wait.
• ModelToComponentFactory.create_rate_limited_multiple_token_authenticator caches instances keyed on the model definition, so all streams share one counter set (same idea as the shared api_budget).

Example manifest usage (source-github):

authenticator:
  type: RateLimitedMultipleTokenAuthenticator
  tokens: "{{ config['credentials']['personal_access_token'] }}"  # comma-separated PATs, or a list
  auth_method: "token"
  quota_status_source:
    type: QuotaStatusSource
    url: "{{ config.get('api_url', 'https://api.github.com') }}/rate_limit"
  quotas:
    - type: TokenQuota
      name: rest
      remaining_path: ["resources", "core", "remaining"]
      reset_path: ["resources", "core", "reset"]
      limit_path: ["resources", "core", "limit"]
    - type: TokenQuota
      name: graphql
      remaining_path: ["resources", "graphql", "remaining"]
      reset_path: ["resources", "graphql", "reset"]
      matchers:
        - type: HttpRequestRegexMatcher
          url_path_pattern: "/graphql"

Changes:

• declarative_component_schema.yaml: new RateLimitedMultipleTokenAuthenticator, QuotaStatusSource, TokenQuota definitions; added to HttpRequester.authenticator and SelectiveAuthenticator.authenticators unions (models regenerated; new classes applied on top of the checked-in file to avoid unrelated codegen churn).
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py: runtime implementation.
• model_to_component_factory.py: create_rate_limited_multiple_token_authenticator with token-string splitting, interpolation, and shared-instance caching.
• Unit tests: seeding/header injection, quota matching, rotation, exhaustion wait/raise, budget throttling, thread safety (no lost decrements across 200 threads; signed token matches decremented token under concurrent rotation), factory token parsing and instance sharing.

Requested by Daryna Ishchenko (darynaishchenko).

Link to Devin session: https://app.devin.ai/sessions/3ced8f6a11d44ed4abc517221294f767
Requested by: Daryna Ishchenko (@darynaishchenko)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783347505-rate-limited-multiple-token-authenticator#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783347505-rate-limited-multiple-token-authenticator

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 143 tests  +16   4 131 ✅ +16   7m 39s ⏱️ -2s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 4283ccf. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 146 tests  +16   4 134 ✅ +16   12m 23s ⏱️ + 5m 8s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 4283ccf. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

[coderabbitai]

Warning

Review limit reached

@darynaishchenko, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e7eb179f-bee7-4d91-bb4d-8027175a1022

📥 Commits

Reviewing files that changed from the base of the PR and between dd9558c and 4283ccf.

📒 Files selected for processing (7)

• airbyte_cdk/sources/declarative/auth/__init__.py
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/spec/spec.py
• unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1783347505-rate-limited-multiple-token-authenticator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}