fix: advance FixedWindowCallRatePolicy windows past all elapsed periods - #1155
Draft
devin-ai-integration[bot] wants to merge 1 commit into
Draft
devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1789429627-fixed-window-rollover#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1789429627-fixed-window-rolloverPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
👉 TL;DR: Fixed-window rate limiting could stop enforcing its call limit after the limiter sat idle across several windows, and rejected calls made at the exact instant a window reset.
Specifically,
FixedWindowCallRatePolicy._update_current_windownow rolls forward across every elapsed period at once and treatsnow == _next_reset_tsas the start of the new window.Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17245:
Related open PRs touching the same class (not merged/conflicting in scope): #985, #989.
Changes
_update_current_window: on rollover, computeelapsed_periods = (now - _next_reset_ts) // _offset + 1and advance_next_reset_tsby that many periods, so alignment is preserved and the pointer lands strictly afternow. Previously it advanced by exactly one period per call, so after N missed periods the next Ntry_acquirecalls each reset_calls_numto 0 and the limit was never enforced.now > _next_reset_tstonow >= _next_reset_ts, so a call at the exact reset instant is admitted instead of raisingCallRateLimitHitwithnext reset in 0:00:00.create_fixed_window_call_rate_policyin the declarative factory is intentionally not touched.Risks
update()sets_next_reset_tsfrom API headers to a time in the past; the factory seeds it 10 days ahead. Behavior change is strictly toward enforcing the configured limit.Test plan
unit_tests/sources/streams/test_call_rate.py::TestFixedWindowCallRatePolicy(freezegun): multiple missed periods enforce the limit; call allowed exactly at reset boundary; alignment preserved after skipping periods; single-period rollover. The first two fail onmainand pass on this branch.poetry run pytest unit_tests/sources/streams/test_call_rate.py -q— 42 passedpoetry run ruff check .,poetry run ruff format .,poetry run mypy --config-file mypy.ini airbyte_cdk/sources/streams/call_rate.py— cleanRequested via the
/ai-fixworkflow on the linked issue.Link to Devin session: https://app.devin.ai/sessions/7d579e79131e44118d81355f44a6b2c6
Open in Devin Desktop: https://app.devin.ai/desktop/session/7d579e79131e44118d81355f44a6b2c6?variant=devin